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ABSTRACT 

Reported is a study on the use of quantitative data 
in evaluating a science course for the purpose of introducing an 
alternative form of information presentation capable of supplying 
qualitative feedback valuable to students, teachers, and curriculum 
developers. Fifty-five teachers, xwa^omly selected during the 1967-68 
Project Physics (PP) experimental period, were classified as the PP 
trial, non-PP control, and PP experieiu^ed groups. Results of the 
pretest and posttest of the Test on Understanding Science and the 
Science Process Inventory tidcen by 1,188 randomly selected students, 
921 in PP. course and 267 in non-PP course, were used in this study. 
Quantitative t-test findings %iere ccmsidered as usable only in 
Speculation of the nature of group differeices. The McMemar chi 
square item analysis was introduced to identify areas of knowl^ge 
and specific ideas for which students ^ow^ a significant increase 
or decrease in understanding. The PP course was found superior to 
non-PP courses in four aspects: science tactics, value of science, 
instruction function of science, and science- society interactioi. The 
increased specificity of qualitative analysis led to high-quality 
feedback. (CC) 
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INTRODUCTION 

This paper reports, on. (1) the limitations to the 
use of quantitative data in evaluating a science course, 
and (2) a complenentary or alternative form of information 
tdiich yields valuable feedback to curriculum developers, 
teachers* and students. This feedback emerges from asking 
such qualitative questions as: What ideas have students 
learned? What misunderstandings have they still retained? 
Harvard Project Physics is partially evaluated in order 
to illustrate the use and limitations of qualitative data. 

r 

A NEED FOR QUALITATIVE DATA 
Traditionally, student achieveaent and course 
evaluation rest firsly upon quantitative data. For 
example, "Students of course E gained S*48 points on 
test T while students of course C gained only 1.71 points* 
Therefore, course E is better than course 

But what does it mean to a curriculum developer 
or teacher for group £ to score 3.77 points more than 
group C? What information does this give him? Unfort- 
unately« it is an ambiguously summative datum which 
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through statistical coaparison leads to a probabilistic 
statement concerning course effectiveness. Most often 
such a statistical statement is misused as an interpret- 
ive and judgemental statement. ^ The education community's 
reliance on purely quantitative data tends to propagate 
this ambiguity and misunderstanding. 

Quantitative data elegantly manipulated with sophis- 
ticated statistics do have their place. Welch and 
Walberg's multivariate analysis of variance study^ is a 
paradigm in this regard. Their multivariate and univar- 
iate F-tests lead to useful coiq)arisons between an 
experimental and control group. Howevr-, because these 
statistical computations rely on total test scores, the 
subsequent cojiq^arisons might very well be cursory. When 
identifying student achievement for the purpose of 
course evaluation, perhaps quantitative data should only 
be used as initial pieces of information. 

Qae of the greatest utilities a test can have is 
siq»plying qualitative feedback to a researcher, teacher, 
and student'.* Thus, one should consider asking quali- 
tative questions such as: What ideas have students 
learned? What misunderstandings have they still 
retained? With the corresponding answers, the relative 
merit of one learning experience over another may be 
considered in less ambiguous terms. 
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Cooley and Klopfer* first demonstrated the 
assets of qualitative data when cvluating Klopfcr's 
History of Science Cases . Klopfer and McCann's^ 
•valuation of a new junior high school science course 
and Klopfer 's^ evaluation of astrononry materials lent 
further credence to the use of qualitative data. 
Itecently, MacKay^ included qualitative data in a sui ey 
report carried out in Australia. The^larity of qual- 
itative data allowed the researchers not only to conclude 
that one group of students 'Tiad learned more** than 
another group, but*also allowed them to describe in 
detail the ideas which the students had generally 
learned. 

The dearth of qualitative data in educational^ 
research points to the pressing need for its inclusion 
in research studies.* Its utility is demonstrated here 
in an evaluation of Harvard Project Physics .* 

ft 

The first commercial edition was published as 
Project Physics . September, 1970; Holt, Rinehart I 
Winston. 



AN EVALUATION OF HARVARD PROJECT PHYSICS* 

A full description of the sample, experimental 
design, and instruments in the study may be found in 
another paper which is being presented to the 46th an- 
nual meeting- of NARST.l Following is a stmusary of es- 
sential inforaation. Fifty-five teachers were randomly 
selected from a total population of physics teachers in 
the United States and Canada. These teachers were then 
randomly assigned to teach Harvard •Proftet Physics . HPP, 
(after having participated in a summer institute) or lion- 
HPP (the physics courses they would have ordinarily 
taught.) An additional group of nineteen teachers, 
experienced at teaching HPP, volunteered to participate 
in the evaluation project. They taught in various regions 
of the United States. A random sample of students of all 
teachers wrote the Test on Understanding Science (TOUS)^ 
or the Science Process Invento ry (SPD^^ on a pretest and 
posttest basis. These instruments do not concern themselves 
with what is normally called "subject matter,^' Instead 

•The evaluation is only a partial one. The student 
learning evaluated in this study is limited by the two 
instruments which are used. To this extent, the evalua- 
tion of Harvard Prcject Physics is limited to knowledge 
about science and scientists. 



they tend to measure one's knowledge about the scienti' 
flc enterprise: the aims of science, its epistciiiology , 
its tactics, its values, its institutional functions, 
its interactions with society, and its human needs. 
There were 921 HPP and 267 non-HPP students. (The exper- 
iaental group was larger than the control group in order 
to amass enough data for nore extensive investigations 
with the HPP students.) 

Results 

The students' responses to the TOUS and SPI were 
analyzed both quantitatively and qualitatively.^^ Accord- 
ing to statistical calculations, t tests for natched pairs, 
the TOUS and SPI scores gained significantly for both 
groups of students. These results are shown in Table I. 
The HPP students' mean gaii» score (7.95 points, TOUS and 
SPI cowbined) proved to be significantly greater than 
that of the norf-HPP students (3.59 points, TOUS and SPI 
combined). However, with this quantitative information 
one can only speculate on the nature of these differences 
between the two groups. That is, the statistical'compar- 
ison of quantitative data appear to 6e grossly uninforma- 



TABLE I 



TISTS FOR SIGNIFICANCE BETWEEN PRETESTS AND POSTTESTS 











SD 


HPP 


TOUS (N-445) 


Mean 


T test* PTob'abllity 




pretest 


34.41 




6.857 








12.31 p < .001 






posttest 


37.54 




7.059 




SPI (N«476) 




• 






prexesr 


1 0 7 « 9 Z 




8.233 








10.65 p < .001 






posttest 


112.34 




8.245 


non- 


TOUS (N»126) 








HPP 












pretest 


35.25 




6.434 








2.83 .01<p<.001 






posttest 


36.42 




6.570 




SPI (N«141) 










pretest 


107.08 




7.789 








4.34 p < .001 






posttest 


109.30 




9.481 



Test for significance of the dixterence between two 
■eans for non-independent saaples. 
•* 

<6D neahs standard deviation. 



- 7 - 



A McNeniar chi square item analysis^ identified 
areas of knowledge and specific ideas for which students 
showed a significant increase or decrease in understand- 
ing. Table II summa rizes the achievement of the HPP 
versus the non-HPP students. A significant increase or 
decrease in correct response between the pretest and 
posttest is indicated by ^ or respectively. The 
test ite«s in Table II are categorised by their content. 
The arbitrary classification scheme is purely a heuristic 
one. 

In four content areas: 

(1) the tactics of science 

(2) the values of science 

(3) the institutional functions within 
science 

(4) the interaction of science with society 
the HPP achievement greatly exceeded that of the control 
group. Within these four categories the HPP students 
significantly improved on forty-three items compared 
with the control group's five. 

Most notably, learning about tactics or methods 
of science appearecTto be HPP's tour de force . All gains 
were accomplished by the HPP students alone. For 
example, a significant number of HPP students learned not 
to think in terms of "the acitntiflc method," while 
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TABLE II 

SUMMARY OF HPP VERSUS NON-HPP PERFORMANCE ON THE 
TOUS AND SPI (ITEMS CATEGORIZED WITH REGARDS TO CONTENT) 



a 



Topic 
Aims of Science 



Epistemology of Science 
Definitions 



Assumptions 



Item^ 

S3C 
S20 
SS6 
T12 
T16 
T23 
S94 
SIS 
S56 



SlOO 

S13 

S40 

S50 

S62 

S89 

S58 

S22 

S70 

S27 

SI 7 

S77 

S114 

S119 

S12 

S126 

S109 

SI27 

S73 

S104 

S44 



HPP 



Non-HPP 



♦ 
♦ 

♦ 
♦ 

+ 



♦ 



♦ 
♦ 
♦ 
♦ 
♦ 



♦ 
* 
* 
+ 
♦ 
+ 
♦ 
♦ 
♦ 
♦ 
♦ 



♦ 



4- 
♦ 



♦ 

♦ 
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TABLE II 
(continued) 



Topic 



Item 



HPP 



Non-HPP 



Tactics of Science 
"The scientific 
method" 



Scientific inquiry 



Values of Science 



S37 


♦ 


S82 


♦ 


S107 




S121 




SI 30 




S135 


♦ 


TS4 


♦ 


SI 15 


♦ 


S133 




SI 12 




S79 


♦ 


S48 


♦ 


T6 




T13 




S38 




T55 


♦ 


S43 


♦ 


S26 




TS8 


♦ 


S88 


♦ 


S93 


♦ 


T26 




T49 


* 


S3 


4- 


S90 


♦ 


S8 


♦ 


SlOl 




S9 


4 


S5S 


♦ 


S102 


♦ 


S97 


* 


S117 


♦ 


S6 




S60 


♦ 



♦ 
♦ 



ERLC 
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Topic 



TABLE 11 
(continued) 

Item 



HPP Non-HPP 



Insti-*-- tnri Functions 
J -r s 



Other fimctions 



Interaction of Science 
With Society 



Human Needs 



T25 
S21 
TS9 

T27 
T2 
TS2 
T28 

T60 
T29 
T4 

T47 

T18 

Til 

T42 

SIO 

T9 

T32 

T31 



♦ 
♦ 
♦ 

■f 

+ 
♦ 

♦ 
♦ 

♦ 
♦ 
♦ 

♦ 
+ 
♦ 



♦ 
♦ 



a ■ 
Included in this swunary are only those Tfsii«! ^nA cot 

cni square value for either the HPP or non-HPP groups. 

The nimber is proceeded bv a "T»» if *Um 4*— 
fro. the TOOS, an ^S" if1ro« tSe SPI 

significant. McNemar chi value f 05 

Sir.!?.: • ^® correct response ,«id "-••for a 
decrease m correct response. 
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non-HPP students apparently remaiaed in their original 
state of misunderstanding. That is, only HPP students 
gained in their understanding that scientists are free 
to use any appropriate tactic in their research, "no 
holds barred" as Bridgman has said. Other topics sub- 
sumed under "tactics of science" included the roles of 
imagination, confirmation, and instrumentation in 
scientific methodology. 

In the content area "values of science," items 
that showed significant improvement encompassed such 
topics as the simplicity value, skepticism, practical 
applications of scientific discoveries, and the values 
Involved in reaching scientific conclusions. Knowledge 
of "institutional functions within science" generally 
increased for the HPP group alone. For exai^le, these 
students became more aware of scientific journals, 
scientific societies, and the international character 
of science. The fourth content area to experience 
significant HPP gains was described as "the interaction 

« 

of science with society." The items dealt with the 
limitation of scientific thought applied in non- 
scientific areas, the influence of govemaeat upon 
scientific progress, and the effect of public support 
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upon science. 

The HPP group significantly gained on twice as many 
items as the non-HPP group in the remaining three areis: 

(1) the aims of science 

(2) the human needs within science 

(3) the epistemology of science including 
definitions, assumptions, general aspects 
of scientific knowledge, laws, theories, 
models, and observations. 

Even though the HPP and non-HPP courses appear more 
s^ilar in these content areas, the HPP course did seem 
to encourage greater achievement. 

Using the data in Table II and referring to the 
pertinent test items, one can co^)are the HPP and non- 
HPP groups in terms of the specific id^as students 
tended to, learn during the year. An illustrative 
complex comparison is made for the secticm "aims of 
science." According to SPI items 20 and 86, most physics 
students regardless of the course of study tended to 
learn that science strives to establish casual relation- 
ships which are not necessarily more complex than prev- 
ious relatipnships . However, only the HPP students siswed 
significant improvement in: (1) learning the aim of 
scientific explanation (TOUS items 12 9 16), (2) dis- 
tinguishing between science and technology (TOUS item 23^, 
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(3) recognizing the importance of prediction is a goal in 
science (SPI item 94) » and (4) realizing that scientists 
do not wish to make prejudiced observations (SPI item 15). 
On the other hand, sone HPP students seemed to be misled 
into thinking that the purpose of experimentation is to 
prove the laws of nature (SPI item 30), while the non- 
HPP students acquired the misconception that scientific 
investigations may be exemplified by the simple mctivity 
of collecting rock<l^ (SPI item 56)* 

Sifliunary 

Compared with other physics courses, the impact of 
HPP on students learning about science and scientists 
appeared to be substantial. The HPP students achieved 
significantly greater gain scores than did the non*HPP 
students. While HPP did not clarify Misconceptions and 
aisunderstandings in the ninds of all students, it did 
tend to inprove (much more than did other physics 
courses) the student's knowledge about science and 
scientists.' In four areas specifically (the tactics of 
science, its values, its institutional functions, and 
the interaction of science with society) the HPP adiieve* 
ment £reatly exceeded that of the control groi^. 
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IMPLICATIONS FOR FURTHER RESEARCH 
Quantitative data do play a logical role in deter- 
mining statistical conclusions for experimental studies. 
However, this role might best be limited to a minor one 
when identifying student achievement for the purpose of 
course evaluation. Qualitative data appear to be far 
more informative to curriculum developers and teachers 
than total score gains and subscore gains alone. In this 
study, the evaluation of HPP demonstrates the increased 
specificity in the results derived from qualitative 
information. Not only can one say th© HPP students 
achieved more than the non-HPP students, but one can 
better understand in what ways (by what knowledge) the 
groups differed. In some measure, the developers and 
teachers of HPP may now understand how their course 
differs or concurs with other physics courses. They can 
realistically anticipate what learning will likely take 
place. If this content is thoiight tp be incomplete, the 
appropriate changes may be made to the course* 

*In addition to comparing experimental and control 
groups, one can also use thf qualitative data to co^are 
the ideas generally learned by students with the expressed 
objectives of the course. Such a study has been coq^leted 
for HPP. 



REFBRENCES 



1. Aikenhead, Glen S. "A New Methodology for Test Construction 

in Course Evaluation." A paper persented at the 
46th annual meetins: o£ the National Association 
for Research in Science Teaching, Detroit, 
March 26-29, 1973. 

2. Heath, Robert W. "Pitfalls in the Evaluation of New 

Curricula." Science Education , 46 (April, 1962), 
216. 

3. Welch, Wayne W. , and Walberg, Herbert J. "A National 

Experiment in Curriculum Evaluation." American 
Educational Research Journal , 9 (Summer, 1972), 
373-383. 

4. Klopfer, Leo, and McCann, Donald C. "Evaluation in 

Unified Science." Science Education , 53 (March, 
1969), 155-164. 

5. Cooley, William W. , and Klopfer, Leo. "The Evalua- 

tiob of Specific Educational Innovations." 
Journal of Research in Science Teaching , 1 
March, 1963), 73-80. 

(. Klopfer, Leo. "Effectiveness and Effects of ESSP ■ 
Astronomy Materials." Journal of Research in 
Science Teaching . 6 (January, 1969), 64-75. 

7. MacKay, Lindsay D. "Development of Understanding 

About the Nature of Science." Journal of 
Research in Science Teaching , £ (January » 
1971), 57-66. 

8. Aikenhead, Glen S. "Hie Measurement of High School 

Students* Knmrledge About Science md Scientists." 
Paper presented at the 44th annual seeting of ^e 
National Association for Research in Sci^ee 
Teaching, Silvti Spriag, ItaTylaad, Nar^ 2S, 1971. 

9. Cooley, Wlllioi, and Kl^fer, Leo* Test en ^dtrstwid- 

iag Science . Prince t^, Jers^: EtBeational 
Testing Service, 1961. 



10. Welch, Wayne W. Welch Science Process Inventory , 
foTP D . Cambridge, Mass.: Harvard Project 
Physics, 1966. 



11. Aikenhead, Glen S. "The Measurement of Knowledge 
iVbout Science and Scientists: An Investiga- 
tion into the Development of Instruments for 
Formative Evaluation." Unpublished Doctoral 
Thesis, Harvard University, 1972, pp. 9S-194. 



